A Closed-form Solution for the Pre-image Problem in Kernel-based Machines

The pre-image problem is a challenging research subject pursued by many researchers in machine learning. Kernel-based machines seek some relevant feature in a reproducing kernel Hilbert space (RKHS), optimized in a given sense, such as kernel-PCA algorithms. Operating the latter for denoising requires solving the pre-image problem, i.e. estimating a pattern in the input space whose image in the RKHS is approximately a given feature. Solving the pre-image problem is pioneered by Mika’s fixed-point iterative optimization technique. Recent approaches take advantage of prior knowledge provided by the training data, whose coordinates are known in the input space and implicitly in the RKHS, a first step in this direction made by Kwok’s algorithm based on multidimensional scaling (MDS). Using such prior knowledge, we propose in this paper a new technique to learn the pre-image, with the elegance that only linear algebra is involved. This is achieved by establishing a coordinate system in the RKHS with an isometry with the input space, i.e. the inner products of training data are preserved using both representations. We suggest representing This is an extended version of the paper [8], winner of the Best Paper Award at the IEEE Machine Learning For Signal Processing workshop. P. Honeine (B) Institut Charles Delaunay (FRE CNRS 2848), LM2S, Université de Technologie de Troyes, 10010 Troyes, France e-mail: [email protected] C. Richard Laboratoire Fizeau (UMR CNRS 6525), Observatoire de la Côte d’Azur, Université de Nice Sophia-Antipolis, 06108 Nice, France e-mail: [email protected] any feature in this coordinate system, which gives us information regarding its pre-image in the input space. We show that this approach provides a natural preimage technique in kernel-based machines since, on one hand it involves only linear algebra operations, and on the other it can be written directly using the kernel values, without the need to evaluate distances as with the MDS approach. The performance of the proposed approach is illustrated for denoising with kernel-PCA, and compared to state-of-the-art methods on both synthetic datasets and realdata handwritten digits.